Video segmentation is the process for breaking a video frame into a background scene and foreground objects. Due to the limitation of computational power, sometimes it is not possible to segment every video frame. Hence, some video frames are not segmented (e.g., some frames are skipped). The conventional process of generating segmentation masks for each the skipped frame includes copying a mask of the un-skipped frames, or through image morphing, which requires the user to manually provide a set of point correspondences between the segmentations of the un-skipped frames. These conventional processes are not practical for many applications, because simple mask copying results in a bumpy appearance, and image morph cannot be performed automatically without manual inputs from the user.